Radiofrequency map creation for data networks

ABSTRACT

A Next Generation Data Network is described. It leverages the “cloud” for data management, frequency data computation and analytics. Training signals are transmitted in a number of different transmit directions and attempted to be received in a number of different receive directions in order to create a radio frequency map of transmit/receive directions that allow a communication path to be created between nodes of the network. The wireless network is a single frequency network that permits limited non-line-of-sight operation.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation of U.S. patent Ser. No.14/214,957, filed on Mar. 16, 2014, entitled “Distribution Node andClient Node for Next Generation Data Network,” which is related to andclaims priority under 35 U.S.C. 119(e) to U.S. provisional patentapplication, Ser. No. 61/789,188, entitled “Next Generation DataNetwork,” filed on Mar. 15, 2013. The contents of these applications arehereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates generally to wireless communication. Inparticular, the present disclosure relates to methods and systems forcontrolling a multi-hop mesh wireless data network.

BACKGROUND

Data networks have traditionally been wired, either copper or fiber.Wireless spectrum has typically been used for subscriber links, not DataInfrastructure. The advent of micro cells, pica cells, nano cells, andhotspots has made the provisioning of wired Data prohibitivelyexpensive. This is a result of inflexibility of wired networks; cablehas to be run to add new points of presence. In many cases, thisrequires that streets be torn up.

Wireless Data networks can be much more flexible, and hence moreeconomically viable. However, existing wireless Data networkarchitectures limit this flexibility in several ways. One common form ofwireless Data networks uses point-to-point radio links, typicallyoperating in the microwave or millimeter wave bands.

Another common form of wireless Data networks use a cellulararchitecture. These networks are based on frequency reuse. Typically acellular reuse scheme where the total available frequency spectrum isdivided into sub bands. “colors”, which are reused in disjointgeographical areas, “cells”. For example, reuse factors of 4, 7, and 12are popular. Other systems reuse time, polarization, or code. Thedisadvantage of all of these fixed, “cellular”, reuse schemes are thatthey are inefficient, resources are wasted. For example, in a 7 to 1frequency cellular reuse system, only 1/7-th of the total availablefrequency spectrum is available in a single cell, even if none of theadjacent cells are being used. Another disadvantage of these frequencyreuse schemes is that it is difficult to add more cells in an area ofhigh demand.

SUMMARY

This invention is a wireless Data network that reuses the same frequencyband in all cells. For example, if the frequency band is divided intothree frequency channels, all three of these channels are available toevery node in the network. Frequency channels are dynamically assignedby the Cloud to maximize network capacity taking into account potentialinterference from both other links in the network and from othersources. This is distinct from previous systems which try to maximizeC/(N+1), ratio of carrier power to combined noise plus interferencepower. Further, the system dynamically adjusts route and frequencychannel assignments, transmit power, modulation, coding, and symbol rateto maximize network capacity and probability of packet delivery, ratherthan trying to maximize the capacity of any one link.

The Next Generation Network is a time domain duplex (TDD) system (it canalso be Frequency domain duplex, however the TDD is the more complexcase, hence explained in detail). All sectors of each node transmit orreceive at the same time. They are synchronized by a master controller.This allows the use of a single frequency across all sectors andmitigates adjacent channel interference. It is a multipoint system wherethere is no limit to how many nodes a node can communicate with, Eachsector on each node has a 2D or 3D beamforming capability. This permitsthe unit to form a high gain beam that is pointed at the destinationnode and packet sent. The receiving node does the same it forms a highgain narrow beam pointed at the transmitting node. Beam coefficients aregrossly computed in the cloud and fine-tuned between the units.

The network uses two forms of route adjustment. A mesh network topologyis used to allow traffic to be routed between two nodes via at least twogeographically independent paths. This provides load balancing, so thatwhen one path becomes congested or degraded, traffic can be routedaround it. It also mitigates the effects of external interference. Thesecond form of route adjustment is “micro-routes”. Rather than justpointing the receive antenna at the transmitter, or pointing thetransmit antenna at the receiver, this invention takes advantage ofmultipath to use indirect (non-line-of-sight) paths in addition to thedirect (line-of-sight) path. The ability to use these paths providesseveral advantages. One is interference avoidance. Any interference thatwas arranging along the main beam will be mitigated by pointing awayfrom it. Another is obstacle avoidance. The micro route can be used tomitigate obstacles, such as trees, signs, trucks, or similar that blocksthe direct line-of-sign path. Further, by pre-computing these reflectedroutes they can be applied in real time.

Frequency channel assignments, transmit power, modulation, coding, andsymbol rate for each link, both direct path and micro route, aredynamically optimized in the Cloud to maximize network capacity. Eachnode transmits probe packets to each of its neighbor nodes that can be nlayers deep using a synchronous scheme. During a time interval know toits neighbors, the node transmits probe packets in various directionswith a known pattern. The neighbor nodes listen for each of thesetransmissions in various directions, and record the received channelmeasurements. These measurements are periodically uploaded to the Cloudand processed. The processing identifies the viable micro routes anddetermines the range of link parameters available for each. Thisprocessing takes into account the received signal strength of the probepackets in each receive direction for each transmit direction, It alsotakes account of the reverse measurements, those made by the other nodewith its receive direction pointed in the previous transmit directionand with first nodes transmit direction pointed in the direction of theprevious receive direction. Data is processed over time to identifystatic routes, those that are available over long periods of time, andperiodic routes, those that are repeatedly available at certain times ofday.

The received signal strength data collected by each unit is sent to theCloud server for processing. This data consists of transmit angles,which in one embodiment of the invention are represented by beamformingparameters, and receive angles, which in the same embodiment of theinvention are represented by beamforming parameters, and received signaltransmit power.

A data processing function in the Cloud processes the data to identifymicro-routes, reflected paths and interference levels between units. Inan alternate embodiment of the invention, the processing is performed ineach unit. In another alternate embodiment, the processing is performedin selected units. For each micro-route, the Cloud processor determinesa set of signal parameters to be used for that route. These parametersare the transmit power level, beam pointing/forming coefficients plusmodulation and coding. In one embodiment of this invention, transmitconstellation pre-distortion parameters are included in the signalparameter set.

The RF mapping function is performed every 60 seconds. In alternateembodiments of this invention, it is performed at different rates. Inone alternate embodiment, it is performed whenever capacity utilizationis low. The RF mapping data is collected incrementally, distributed overtime in different sections of the network. In another embodiment of theinvention, the two units at the end points of a micro-route negotiateevaluations of beam pointing perturbations to optimize performance overtime.

Meteorological data (weather forecasts) of rain and wind storms is usedto identify phenomena, which might impact RF link performance. When alink degrades due to rain, the meteorological data is used to predictthe motion of the rain microcell, These predictions are used to routetraffic around the rain microcell to accommodate the reduced linkcapacity.

In high wind conditions, or when the structural mount is flexible, thebeams are widened to maintain lock when the antennas are moved by thewind. To compensate for the widened, lower gain beams, the Cloud alsoreduces the data rate on these links. Wind direction is used by theCloud to anticipate the need for beam widening.

Coarse beam parameters are computed on the server. These are geometricin nature and downloaded to the unit. Further the server can alsocompute perturbations on these parameters that the unit can try and thencategorize by performance.

The Next Generation Network mitigates external interference is severalways:

1. By changing frequency channel

2. By using a reroute (route diversity) to avoid an interfered link

3. 13u using a micro-route to “look away” from the interference

4. By using a spatial interference canceller

5. By using a spatial interference canceller and a micro-route if theinterference is along the original path or if additional interferencesuppression is required

6. By using adaptive modulation and coding

7. By using adaptive modulation and coding with any of the othertechniques

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a topology of the next generation data network, inaccordance with one embodiment of the present invention.

FIG. 2 shows the deployment of the distribution nodes (DNs), clientnodes (CNs), and a cloud controller (CC) in the next generation datanetwork, in accordance with one embodiment of the present invention.

FIG. 3 shows the four sectors of a distribution node, in accordance withone embodiment of the present invention.

FIG. 4 shows the next generation data network split into subnets eachassociated with a subset of fiber connected nodes, in accordance withone embodiment of the present invention.

FIG. 5 shows a transmit/receive exception pair characterized by the RFmap function as an undesirable path in the next generation data network,in accordance with one embodiment of the present invention.

FIG. 6 shows a micro route around a physical or interference blockage inthe next generation data network, in accordance with one embodiment ofthe present invention.

FIG. 7 shows a re-route around a physical or interference blockage inthe next generation data network, in accordance with one embodiment ofthe present invention.

FIG. 8 shows interference suppression using a micro route in the nextgeneration data network, in accordance with one embodiment of thepresent invention.

FIG. 9 shows the beamforming characteristics of a spatial interferencecanceller used in the next generation data network, in accordance withone embodiment of the present invention.

FIG. 10 shows packet switched beams transmitted by a distribution nodeof the next generation data network, in accordance with one embodimentof the present invention.

FIG. 11 shows the block diagram of the four sectors of a distributionnode of the next generation data network, in accordance with oneembodiment of the present invention.

FIG. 12 shows using beamforming to maximize the coverage region of theside lobes for communication in the next generation data network, inaccordance with one embodiment of the present invention.

FIG. 13 shows the elements of the antenna arrays of a sector of adistribution node or a client node in the next generation data network,in accordance with one embodiment of the present invention.

FIG. 14 shows the block diagram of a sector of a distribution node ofthe next generation data network, in accordance with one embodiment ofthe present invention.

FIG. 15 shows a bus interconnection between the sectors and fiber 1/0 ina distribution node of the next generation data network, in accordancewith one embodiment of the present invention.

FIG. 16 shows characterization of the RF environment for reflectionusing RF maps in the next generation data network, in accordance withone embodiment of the present invention.

FIG. 17 shows data collection of transmit angels, receive angles, andreceived power from the transmitter and receivers of a next generationdata network for generating RF maps, in accordance with one embodimentof the present invention.

FIG. 18 shows prediction of rain microcell to reroute traffic around therain microcell in the next generation data network, in accordance withone embodiment of the present invention.

FIG. 19 shows beam widening to maintain lock on a particular link of thenext generation data network, in accordance with one embodiment of thepresent invention.

DETAILED DESCRIPTION

The Next Generation Data Networks topology is shown in FIG. 1. Thenetwork 100 (FIG. 2) consists of Distribution Nodes (DNs), Client Nodes(CNs), and a Cloud Controller (CC), The Distribution Nodes (FIG. 3) mayconnect to a fiber point of presence to get or send data from the“cloud”, The distribution nodes can talk with other distribution nodesand client nodes, The DN has one to four sectors 302, 304, 306, 308. TheClient Nodes have one or two sectors. The Cloud Controller managesrouting and network performance analysis. A network may have multiplefiber connected nodes that can be used for redundancy and/or todistribute traffic entry/exit points, These networks can be furthersplit into subnets 402, 404 each associated with a subset of fiberconnected nodes (FIG. 4). These subnets are time synchronized tominimize interference and edge effects between the subnets. Frequencychannels may also be managed between these subnets to minimizeinterference between the subnets.

The Cloud Controller computes all of the routes from each DistributionNode to each of the Client Nodes and generates routing tables for eachDistribution Node to/from each of its Client Nodes. These routing tablesare downloaded into the nodes. The tables tell each node how to route apacket to its destination. Each node has its own MAC address. The cloudhas a bird's eye view of the network as it collects statistics on eachlink to characterize it. These statistics are (but not restricted to)packet loss, obstructions and signal attenuation, structure rigidity,available of single or multiple micro routes. Using this data itcomputes predicted probability of packet delivery for each MAC addressserviced by that DN. Another option is for the CC function to downloadall routes to the fiber connected DNs, and to each DN—incremental routesto destination MAC addresses and micro routes relevant to that DN. Anexample of the schedule is:

Macro Routing (once/10 sec)

Cloud computes all possible routes between fiber nodes and MACaddresses;

Uplink & downlink;

Cloud Ranks these routes;

Routes are weighted with link quality, speed, time of day, and trafficcongestion;

For each MAC address we have primary and several alternate routes thatare pre-computed with a cost function;

Cloud refreshes ranking—no faster than once every 10 seconds;

Refresh triggered by a substantial change in link quality;

Refresh includes transmit parameters of links;

Route probe commands;

The cloud keeps alternate routes alive with pings and pre-scheduledslots.

Micro Routing (once/hr.)

Cloud refreshes micro routes & ranking every hour based on RF mapupdates & Route Probes.

Exception Pairs (once/hr.)

Exception transmit receive pairs for SFNs are recomputed every hour.

The other functions performed by the cloud controller is creation of RFMaps, management and command of data collection by nodes, analyses oflink performance data collected by the nodes, determination of transmitpower levels for the nodes and clients that are dependent on thedestination of the packet. The cloud controller also performs managementof the single frequency network, virtual networks, sub nets, andcoexistence with other networks. The cloud controller may also manageinterference at the node level or have the fiber connected node and thenode itself manage these functions depending on timeliness of a responsefrom the cloud controller in response to changing conditions.

When a fiber connected node receives a packet, either from the fiber fora Distribution node, or from a client for any node, it finds thedestination MAC address in its routing table, and appends the routingheader specified in the table to that packet. The packet is thentransmitted to the next node in the header. The header entry tells thenode which sector and beam number in that sector to use. Prior totransmission the sending node strips the header entry it just used fromthe packet to expose the next entry which will be used by the receivingnode.

To implement load balancing, the Cloud Controller includes two, or more,entries in the routing table for a given destination node. Each entryhas an assigned probability. The sum of the probabilities for a givendestination is always one. Thus if there is only one entry, the assignedprobability value is 1. If there are more than two entries for a givendestination, the sending node generates a random value between 0 and 1.If the value is less than the probability assigned to the first entry,that entry is used. If not, the value is compared to the sum of theprobabilities assigned to the first two entries. If the value is lessthan this sum, the second entry is used. If there are more than twoentries, this procedure is repeated until the packet is sent. Further,the rules or cost function used for routing can consist of (but notrestricted to)

No Exception Routing Pairs (SFN)

For SFN operation we create transmit and receive exception pairs (FIG.5) derived by the RF Maps function. That is, if a transmitter is sendinga packet to a client/DN with a specific beam type (main or micro route),that may interfere with other units. We ensure if these units arereceiving at that time that their main beams are not pointed in theinterference direction. This can be achieved in two ways one is leavethe receive slot empty or option 2 (the preferred option) is to orderthe packets so that the receiver and transmitter have their beamsoccupied in different directions. Exception pair may also be created toprevent overlapped transmission beams of two adjacent sectors of the DNfrom interfering with one another.

Only Terminal Traffic on Low Reliability Links

On low reliability links where the net data throughput is low we maywant to restrict traffic over those links to traffic where thedestination MAC addresses are only reachable by going over those links.Further, we leverage the fact that traffic density on a link decreasesexponentially on average as we move away from the fiber connected node.This gives us excess unused capacity on most links that are 1 or morehops away from the fiber connected node.

Lower Modulation/Higher Error Coding on Low Reliability Links

We can further enhance the performance of these links by slowing downthe payload rate to ensure timely packet delivery by reducing themodulation complexity (qam16 to qpsk for example) for example and/orincrease the error coding (Rate ¾ to rate ½ code for example).

Lower Modulation on Sparse Data Traffic Links

On links with sparse data that is a lot of unused capacity we can reducethe modulation and increase the error coding to achieve a higherreliability of packet delivery.

Double Packet Routes of Trunk Data Over Lossy Links

High priority packets and “trunk” data (that is data that needs to beforwarded on) can be sent over multiple routes simultaneously. Furtherlarge packets can be broken down into smaller packets as that maypreserve the integrity of the packets that are being received fromdifferent directions.

Maximization of Probability of Packet Delivery

The above mentioned technique may be used in addition to algorithmchoosing links for the route that maximize the probability of packetdelivery, The fiber connected node and the cloud routing & processingfunction are compiling statistics on each link that are being used topredict future performance of each link.

A Minimization of Hops

If the probability of packet delivery is the same on two routes, andthere is no congestion, the route with the lower number of hops will bechosen.

The Next Generation Data Network 100 implements two forms of routeadjustment. The network topology allows packets to be routed between twonodes via at least two spatially independent paths 702, 704. Thisprovides load balancing, so that when one path becomes congested ordegraded, traffic can be routed around it. It also mitigates the effectsof external interference. The second form of route adjustment is“micro-routes” 602. Rather than pointing the receive antenna at thetransmitter, or pointing the transmit antenna at the receiver, thisinvention takes advantage of multipath to use indirect(non-line-of-sight) paths. The ability to use these paths providesseveral advantages. One is interference avoidance. Any interference thatwas aligned with the main beam is mitigated by pointing away from it.Another is obstacle avoidance. The micro-route 602 can be used tomitigate obstacles, such as trees, signs, trucks, or similar objectsthat block the direct line-of-sight path.

Frequency channel assignments, beam forming coefficients, transmitpower, modulation, coding, and symbol rate for each link, both directpath and micro-rout; are dynamically optimized by the Cloud Controllerto maximize network capacity. The two nodes have the option of furtheroptimizing the cloud computed beam parameters to include multipatheffects, unit misalignment, calibration & temperature dependent errors,etc.

Each node transmits probe packets to each of its four neighbor nodesusing a synchronous scheme. During a time interval know to itsneighbors, the node transmits probe packets in various directions with aknown pattern. The neighbor nodes listen for each of these transmissionsin various directions, and record the received channel measurements.These measurements are periodically uploaded to the Cloud Processor andprocessed. The processing identifies the viable micro-routes anddetermines the range of link parameters available for each. Thisprocessing takes into account the received signal strength of the probepackets in each receive direction for each transmit direction. It alsotakes account of the reverse measurements, those made by the first nodewith its receive direction pointed in the previous transmit directionand with second nodes transmit direction pointed in the direction of theprevious receive direction. Data is processed over time to identifystatic routes, those that are available over long periods of time, andperiodic routes, those that are repeatedly available at certain times ofday.

To mitigate the impact of lossy links, retransmissions are used. Thereceiving node is capable of receiving fragments of a packet over timeon the same link or distributed over other links and reassembling them.When lossy links are detected, the sending node fragments the packet andsends copies of fragments using different routes to achieve routediversity and increase the probability of packet delivery. The node cantradeoff the losses between all of the entries in the routing table fora given destination node, including the direct-route and a micro-route.If the losses on the micro-route are lower it may choose to use themicro-route for packet delivery.

For a link that is unresponsive, due to a physical blockage or radiointerference, on the path, the transmit node tries to establishcommunication on a “micro-route”. It may try multiple micro-routes. Ifthat fails it returns the packet (or packet number) to the last unit(DN_(n-1) unit) that sent it.

For rerouting, FIGS. 6 and 7, the node has a list of alternate routes702, 704 for each destination node pre-computed by the Cloud Controller.The node modifies the routing header using one of these and retransmitsthe packet. It is entirely possible that the packet (or packetidentifier) may retrace several steps before it gets on a new route.

Further, this unit deletes routes that contain the failed route from itslist. This is also done at the fiber connected node and all other nodesin the network. Only the cloud computation can reestablish this route.The deletion can be done based on persistence or on just one occurrenceof failure.

All communication failures, link SNR etc. are communicated back to thefiber node and the cloud.

Each sector of the node is equipped with an FFT, which runs over theentire spectrum creating a power spectral density map or a “spectrumanalyzer”. This data is used by the Cloud Controller to select thefrequency of the link. The transmit and receive frequencies can bedifferent and are determined by the interference at the receiver ratherthan at the transmitter. Further each sector of the node can operate ona different frequency.

For each fiber point or broadband point of presence (a high bandwidthlink for taking and bringing data from carrier/internet) and eachdestination MAC address on the network, the Cloud pre-computes thepossible routes with a cost function. The net of the cost function ischaracterized in data rate, delay, and reliability of packet delivery.It also takes into account the availability of multiple routes betweentwo units using micro routes.

Once a unit determines it cannot communicate on its primary path itutilizes pre-stored micro routes or an alternate path computed by theCloud. If these fail it sends the packet back to the DN_(n-1). DN_(n-1)has a list of pre-computed routes for a destination mac address.DN_(n-1) replaces the packet routing header with one of the pre-computedroutes 704 that does not utilize a “failed” route 702.

The fiber connected node and the server are informed of the failed routeor the use of micro-routes and any other route anomalies. They are alsogiven information on links that are operational on signal level, signalheadroom, fading margin etc.

The Cloud computes all routes from the Distribution Node to thedestination node. These are downloaded into the fiber connected unit. Inan alternate embodiment of the invention, they are downloaded into anauxiliary units attached to the fiber connected unit. When the fiberconnected unit receives a packet for a MAC address it adds the routingheader on it, and sends it to the appropriate adjacent unit. The routingheader tells a unit as to which sector to transmit the packet, and thebeam “number” on that sector. The unit strips the parameters from theroute header exposing the next one for the receiving unit. The fiberconnected unit will round robin through equal cost or near equal costroutes for load balance on the network.

If a communications failure occurs, the node first retransmits thepacket. If that fails, it tries a micro-route or an alternate routecomputed by the Cloud. If these fail, it returns the packet to the lastunit (DN_(n-1) unit) that sent it. The DN_(n-1) unit has a list ofalternate routes for each destination node, pre-computed by the CloudController and downloaded to that unit. It modifies the routing headerusing one of these and retransmits the packet. This process is repeateduntil the packet is either returned to the Distribution Node, or it isdelivered. If it is returned to the Distribution Node, that node resendsit on a different path.

When a failure occurs, the sending unit may delete the failed primarypath, the alternate paths, and/or the micro-route from its routingtable, and propagates this information to the other nodes in thenetwork, including the fiber connected node. The sending unit may alsorestrict the use of the primary path, certain alternate paths, and/orthe micro-route between the updating of the routing table by the Cloud.The fiber connected node reports this information to the Cloud. Only theCloud processing can reestablish this route. In an alternate embodiment,instead of deleting a micro-route based on a single failure, a failurecounter is maintained for each micro-route, and only when a predefinedthreshold is exceeded, is the micro-route deleted.

The means of detecting a static reflector are to collect reflection dataat different times of day or days of week or month or year. If thereflected path appears in all of them accounting for the variance ofbeam pointing errors, it is deemed to be a static reflector. Dynamicreflectors that have a known periodicity may also be used but as afunction of that periodicity.

By using a micro-route, rather than the direct path, the Next GenerationNetwork changes the arrival angle of the signal at the receiver. Thisallows the network to maximize the signal to interference ratio at thereceiver. For example, typical side-lobe to main-lobe gain is −13 dB,and a typical micro-route loss is 3 dB. In this case, the improvement inC/I is 10 dB. (FIG. 8)

Micro Route+ Modulation

In this case we use the microroute and reduce the modulation & effectivedata rate on the link. For example if we increased the margin by 6 dBfrom modulation-data rate then our effective margin over the interfereris 16 dB.

Micro Route+ Interference Cancellation

We use a spatial interference canceller (FIG. 9). The canceller does an“inverse beam former” or creates a null in a direction. We have shownthat these nulls maybe as deep at 28 dB over and above antennaisolation. In this case we have an effective margin of 38 dB over theinterference.

Micro Route+Interference Cancel+Modulation

In this case if we had a 6 dB advantage from modulation-data rate oureffective margin is now 44 dB.

Single Frequency Networks

We enable the complete network to be run on a single frequency channelrather than alternating frequencies. This enables the most efficientusage of the frequency spectrum. We achieve this via RF Maps. Using theRF Maps we

1. Compute Power levels for the unit to transmit a packet (knowing itsdestination). Computed power levels reflect the real world power that isrequired to close the link at a desired reliability/availability level.It is important not to transmit at max power where we can use lowerpower to minimize self interference in the network.

2. Beam pointing coefficients coordinated with the packet transmissionsequence across exception pair links. This eliminates the “exceptionpairs” generated by the RF Maps function

3. Synchronization and synchronization offsets for irregulardeployments. In a regular or symmetric deployment alternate nodestransmit and receive. In an irregular deployment there may be multipleDNs on a path corresponding to one link on another path that need to besynchronized.

4. Managing C/I (Carrier to Interference ratio) at nodes via transmitpower, modulation, packet sequence and beam pointing coefficients. TheC/I is managed so that the packet can be delivered with a reliabilitylevel that has been guaranteed to the user.

5. Use 3D RF maps to compute off axis self interference combinations

a. Auto measure sidelobe performance;

b. Utilize the self interference cross correlation matrix to eliminatetransmit receive combinations (beam weights);

c. Achieved via packet scheduling.

Virtual Networks

Multiple virtual networks can be created over the same hardware links.These networks can each be assigned data bandwidth at the fiber node(s).The reason is that the fiber node determines total throughput of thenetwork. Further, we can overbook each network if required usingstatistics that are available historically and that we have collected onthe nature of the burstiness of the data. This will allow us to allocatebandwidth dependent on the guaranteed level of service for each virtualNetwork.

Network Coexistence

Multiple Operators may plan to use our equipment in the same area, andmaybe on the same poles. Using communication between our cloudcontrollers we can manage the frequency, the synchronization of transmitand receive slots to minimize interference and maximize throughput.Further we can also coordinate packet sequencing between these networksto maximize antenna isolation both from beamforming and polarization.The SFN is key to making these networks operate as it increases thedegrees of freedom that we have to operate both in frequency and/orpolarization.

Network and SubNet Synchronization & Security

The network uses two levels of synchronization. Coarse synch using GPSto a nominal of 100 nanoseconds and a fine synch that using signalingbetween units. Sub nets are synchronized in the cloud and theirtransmit, receive, and beam pointing along the edges of subnets iscoordinated and synchronized. Further subnet coordination also utilizesthe sparse data or low data densities of the links at the periphery ofeach subnet to maximize data traffic in the core of each subnet. Thecore is defined as the fiber DN and the first hop DNs. Securityleverages on the delays between units and time of travel between units.Using this as a key we “open” the packets only at the destination.Further due to the assignment of different routes along which the packetmay travel, assigned at the fiber attached node the key is randomized.Further error bounds are placed to account for retransmits that maybeencountered along the way. Alternatively we can add “incremental” keysin each DN to account for retransmits.

Distribution Node (FIG. 3)

Describes nodes in a millimeter wave multi-hop network that provide highdata rates (of the order of multiple Gigabits per second), flexibledeployment (not requiring manual aiming of antennas) and robustness tomultipath fading. FIG. 3 shows a node with 4 faces or sectors 302, 304,306, 308. All sectors transmit and receive at the same time. Each sectorhas a packet switched beam. That is the beamforming coefficients areapplied to enable the sector to point the beam at the client or receiverof the packet to be transmitted. Further, the beam 1002 is then switchedto point (FIG. 10) in the direction 1004 of where the next packet needsto be sent. Each sector has a table of MAC or IP addresses andbeamforming coefficients that the sector is authorized to communicatewith. There may be multiple sets of beamforming coefficients for themain beam or a range assigned to each coefficient. Micro routebeamforming coefficients are also included in this table. All 4 sectorsare synchronized, one sector is the master and the other 3 sectors arethe slaves (FIG. 11). All sectors transmit OFDM signals. However, thesystem can work with single carrier signals.

The nodes can perform adaptive transmit and receive beamforming, andprovide omnidirectional coverage as well as high antenna directivity,That is, each node can communicate with neighbors in any direction ofits choosing in a 360 degree field of view, while being able tosynthesize a highly directive beam in the direction of the node that itis communicating with at a given time. Link robustness is achieved byproviding diversity against fading, which can occur due to multipathreflections from surfaces such as the ground, building walls, orvehicles. The throughput available at each node is increased by allowingfor multiple simultaneous links with different neighbors.

The reason for multi-hop/mesh is that it permits us to “look” aroundcorners/buildings and extend the distance we cover. By having multipleroutes to the destination we also increase the effectiveavailability/reliability of the network.

There are two node types, distribution nodes (DNs) and a client node(CNs). A distribution node has up to four sectors, each covering 6degrees×90 degrees (we can change this coverage for 2D beamformingcases), The fundamental beam is approximately 6 degrees×3 degrees and isbeam formed to sweep over the complete angle. Each sector has a beamthat can be pointed +/−45 degree. Further, we can reduce the coveragearea but utilize the sidelobes to communicate with the close in clientswho maybe on the extremities. For example (FIG. 12), we can reduce theactual antenna coverage to 6 degrees×45 degrees, and use the beamformingto maximize the coverage region (that is maximize the side lobes ratherthan main lobe). That is clients in the 22.5 degrees on either side ofthe coverage region are communicated to by the sidelobes. These linkscan be closed at a lower transmit ElPR as the clients are closer inand/or at a lower data rate. This enhances the link for the “trunk”connections. These nodes are reconfigurable for other angular coverage,which can include beam forming in elevation in addition to azimuth.

In an alternate embodiment, a two dimensional (2D) sweep is used. This2D beam forming has an n×m degree beam that can be steered vertically inelevation and horizontally in azimuth. This embodiment is used forbuilding top to building top as buildings are at different heights. Itis also used to illuminate the side of a building from the street or topof another building.

The antenna array has 32 elements, each 6 degree by 90 degree(FIG. 13).In alternate embodiments, it may have 16, 12 or any other number ofelements. There are two such arrays per sector. On transmit, both arraysare used with an Alamouti code that permits mitigating multipath effectsand increases the transmit power via combining. On receive, both arraysare used with MRC combining.

A block diagram of the sector is shown in FIG. 14. It consists of theantenna arrays, which feed into the beamforming chips. These chips canbe n element chips for simplicity we describe a 32 element beamformer.These beamformer chips use the coefficients for a packet on transmit orreceive. We assume a single transmit/receive chip for each antennaarray. We can have a separate transmit and receive chip with a switch orseparate arrays for transmit and receive. These chips can output/inputthe combined RE signal at baseband or an IF frequency. This signal ismodulated/demodulated by the communication processor which filters,combines signals from the two antennas for receive or generates twosignals on transmit. It also has a QAM/QPSK mod/demod function witherror coding. Other functions are shown in the FIG. 14 as an example.The MAC function is part of this processor it can be in the basebandchip or a separate FPGA. Further Mac filtering takes place that is therouting header is looked at to determine where it goes and is sent tothe appropriate sector directly or via the primary sector. We can usethe 10G bus as the interconnect bus or multiple 10 buses or some otherdigital bus between the sectors. Connection to fiber 1/0 is shown inFIG. 15.

The multipath propagation characteristics for a highly directive mm wavelink are quite sparse. Spatial diversity is provided by using multipletransmit and/or receive arrays offset in the horizontal and verticaldirections. The horizontal offset provides diversity against wallreflections, while the vertical offset provides diversity against groundreflections. The amount of offset is chosen based on typical link rangesand typical distances from scattering surfaces. Thus, each receiveantenna array would generate a single mm wave signal. For diversitycombining of signals from multiple receive arrays, these signals can befurther combined by adjusting gain and phase (e.g., based on a maximalratio combining strategy), or one of the signals can be selected (e.g.,based on amplitude or power). This can then be sent to the singlechannel chipset.

Each node has a centralized controller that routes packets to and fromthe appropriate faces. Isolation between the radio frequency (RF)circuits for different faces is sufficient that all faces cansimultaneously transmit, or simultaneously receive. By designing higherlayer protocols appropriately, the need for a face to transmit whileanother face is receiving can be avoided, thus greatly simplifying theisolation requirements.

An additional means of increasing data rate and interference rejectioncould be to use polarization coding and reverse polarize the twoantennas in the sector. We could take this approach for doubling theeffective throughput l:0 from the unit.

In one embodiment, output from the unit is GigE, SynchE; power into theunit is 48V DC, 110 VAC, 240 VAC; and interconnects between the sectorsis via a multiple gigabit Ethernet ports or a single 100 port.

Other Functions and Components in the System

1. 3D RF Maps

We need to characterize the RF environment for reflections (FIG. 16),interference (FIG. 5), and link losses. Characterization is done usingprobe packets. While we discuss a two dimensional case, the maps aregeneralized for three directions. With vertical scanning (or elevationscanning on the beam) we can characterize the reflectors, interferenceand side lobes in three dimensions. For simplicity of understanding wediscuss a two dimensional case.

To do this we start with a transmit beam on unit A 1702 and receive onall “surrounding” units 1704, 1706, 1708. The number or depth ofsurrounding units is based on the propagation losses. Data collection isperformed as follows (see FIG. 17).

Unit A1 1702 transmits at angle i. All units 1704, 1706, 1708 receive atangle j. And then at j+1, j+2, . . . n;

Unit A1 1702 then transmits at angle i+1, units receive at j, j+1, . . .n;

Unit A1 transmits at angle m, all units receive at j, j+1, . . . n;

Unit A2 transmits in the same sequence as A1 (above), all units receivethe A2 signal at j, j+1, . . . n;

Unit Ak 1710 transmits at angle m, all units receive the Ak signal at j,j+1, . . . n.

The data collected on transmit receive are transmitted to the server inthe cloud. The data consists of transmit angle or beamforming parametersand receive angle or beamforming parameters, received signal transmitpower, multipath parameters if any.

Note transmit and receive signals are shown as “geometric” forsimplicity of explanation; however these beams may be “amoeba shaped”.For example, amoeba shaped beams may be constructed from a combinationof geometric beams from the multiple antenna elements (e.g., 32) of theantenna array. The amoeba shaped beams may be used to increase theprobability of packet reception or to increase the date rate viaindependent paths. In one embodiment, amoeba shaped beams may be createdbetween two nodes independent of the Cloud.

The data analytics function in the server or super unit or possibly ineach unit, sorts through the data to compute reflected paths andinterference between units. For each of these a signal power level isassociated, a modulation parametric set is associated which may includedistortion of the QAM constellation etc.

This data collection exercise can be performed on a periodic basis or adhoc. The data collected can be incremental, that is we do not have tocollect all data in one contiguous set but can be distributed over time.Further we will re-explore the reflected paths periodically, further thetwo units using the reflected path may further optimize beam parametersbetween them to optimize communication.

Whenever there is no traffic or periodically the units are set to scanone transmitting and one or several receiving. We correlate the transmitpower and “angle” with what receivers see again, received power andangle. The angles can be in azimuth and elevation.

To make a single frequency network we use the RF Maps to pin pointinterference between cells. As both “cells” have beam forming we create“packet exclusions,” that is if a packet is being sent to a client thenthe interfered node will not look (receive/transmit) in that directionwhile this operation is in progress.

RF Roads are angular Transmit—receive combinations. Discover RF roadsbetween units using the scanning beams;

RF Roads have data speed, reliability, and exclusion zones;

For desirable roads—permanent reflectors are sourced by collecting datarepeatedly over time.

Weather & Environment Tool

Using Met data we are aware or rain & wind storms 1802 that may degradethe performance of our links. When we see a link degrade due to rain wecan predict the motion of the rain microcell via met data (winddirection & speed). This prediction is used to route traffic around therain microcell and link capacities in the rain micro cell will bereduced (FIG. 18).

For wind we collect data on degradation of the beam—that is we may haveto widen a beam to maintain lock on a particular link. We can use thisdata that we have collected historically, to predict effects in a windstorm which will require that we predict and widen the beam 1902, 1904(FIG. 19), and reduce the data rate to maintain the link budget. Furtherwe can use wind direction in the storm to predict specifics of beamparameters and how they are affected. (Beam may only need to be widenedfor particular wind directions).

3. Beam Parameter Computation

Coarse beam parameters are computed on the server. These are geometricin nature and downloaded to the unit. Further the server can alsocompute perturbations on these parameters that the unit can try and thencategorize by performance. Initial beams maybe computed using the knownGPS locations of the two end of the link and approximate direction ofthe sector face. By using multiple measurements to various destinationsthat the sector can communicate with, the Cloud Controller can computethe “nominal” direction of the sector (that is at 0,0).

When a new unit is added to the network its geographic location isentered into the provisioning system. The appropriate DNs and theirappropriate sectors (using GPS data) will look for the new unit. The newunit is in receive & scan mode unit it acquires the invitation to jointhe network from a DN. AES keys maybe exchanged before it is permittedto enter the network. Further, it will receive the definition oftransmit slots, etc., from the DNs.

The perturbations can be for enhanced performance for multipath(reflectors have been characterized in RF Maps) or for interferencecancellation, or both. The interference cancellation nulls can bepre-computed based on beam directions and cancellation nulls in otherdirections. Perturbations for antenna imperfections are also taken intoaccount and calibrated in the Cloud Controller. This is similar to theprocess to calibrate misalignment or the mounting angle of the sector.

Transmit or receive beamforming can be performed based on any of anumber of well-known mechanisms, including beamforming at radiofrequency (RF) or intermediate frequency (IF) by controlling therelative gains and/or phases of each element, or beamforming in complexbaseband by controlling the complex gains of the baseband signal to/fromeach element for transmit/receive beamforming.

The beamforming and diversity functionalities can be modularized andimplemented in a number of ways. As one example, sophisticatedintegrated circuits developed for short range (e.g., indoor)applications can be leveraged for the longer range outdoor applicationsconsidered in this disclosure. As FIG. 14 show, a single channel chipsetthat handles up/down conversion, baseband processing, and linklayer/medium access protocols, can be used for each face. Alternatively,Beamforming chipsets customized for the outdoor application can belayered on top of single channel RF up/downconvertor chips. In order tocontrol line losses, it is advantageous to place a beamformingintegrated circuit (IC) close to the antenna array that it iscontrolling. For transmission, a transmit beamforming IC would take themm wave signal from the single channel chipset. replicate the signal foreach transmit element, provide power amplification, and modulate thegain and phase to beamform in the appropriate direction. The amount ofpower amplification would be set so as to adhere to regulatoryguidelines on radiated power, while overcoming line and coupling lossesfrom the single channel chipset to the antenna elements. For reception,a receive beamforming IC would provide low noise amplification as closeto the antenna elements as possible (so as to improve the overall noisefigure), while adjusting the gain and/or phase of the received signal ateach element before summing these signals to implement receivebeamforming.

4. Routing Tool

For each fiber point or broadband point of presence (a high bandwidthlink for taking and bringing data from carrier/internet) and eachdestination MAC address on our network, we pre-compute the possibleroutes with a cost function. The net of the cost function ischaracterized in data rate, delay, and reliability of packet delivery.It also takes into account the availability of multiple routes betweentwo units using micro routes.

Further, for rerouting/routing failures, once a unit determines itcannot communicate on its primary path it utilizes pre-stored microroutes or an alternate path from the routing table. if these fail itsends the packet back to the DN_(n-1). DN_(n-1) has a list ofpre-computed routes for a destination mac address. DN_(n-1) replaces thepacket routing header with one of the pre-computed routes that does notutilize a “failed” route.

The fiber connected node and the server are informed of the failed routeor the use of microroutes and any other route anomalies. They are alsogiven information on links that are operational on signal level, signalheadroom, fading margin etc.

Routing is computed based on RF parameters/ statistics of each link andtraffic on the link, to maximize probability of packet delivery at itsdestination. The routing tool can also order increased time coding ofpackets for error recovery, usually applied to high priority/criticalpackets and also for lossy links, For destinations that can only bereached via excessively lossy links the routing tool may order two ormore copies of the packet to be sent on distinct routes, where thereceiving client has the responsibility to reconstruct the originalpacket from error corrected fragments of the original packet. Furtherfor excessively lossy links the routing tool may order a large packet bebroken down into smaller sub packets.

Routing header may include:

Route number, sector number, beam number, power to transmit at,modulation and error correction parameters, number of bits;

Route number+ 1 , sector number, ... ... Ethernet port address; Data;... Data.

5. Micro Route;

Micro-route is a reflected route, pre-computed in the cloud using “RFMaps” The parameters (beam angle, modulation, coding, tx power, . . . )for these routes are downloaded to the unit and then fine-tuned by theunit. Micro routes utilize “static reflectors” who properties are knownas a time of day or are static over all time.

The means of detecting a static reflector are to collect reflection dataat different times of day or days of week of month or year. If thereflected path appears in all of them accounting for the variance ofbeam pointing errors, it is deemed to be a static reflector. Dynamicreflectors that have a known periodicity may also be used but as afunction of that periodicity,

Micro route information also includes beam parameters, data rate,modulation, error correction and other parameters to let the units knowhow to close the link at a required reliability level. Multiple microroutes between two units can be downloaded and prioritized.

The other usage for micro routes is for interference management. Bychanging the angle of arrival of the signal it can present a side lobeof the antenna gain pattern to the interference. Micro routes areperiodically tested along with all other routes by the Cloud Controller.

I claim:
 1. A method, comprising: configuring a transmit beam of a firstnode in a wireless network to transmit a signal at a plurality of firstangles; while the first node is transmitting the signal at each firstangle in the plurality of first angles, configuring a receive beam of atleast one other node in the wireless network to scan a plurality ofsecond angles; and configuring the at least one other node to measuresignal parameters of the signal at each second angle in the plurality ofsecond angles; and calculating an RF map based at least in part oncharacteristics of the transmit beam at each first angle in theplurality of first angles, corresponding characteristics of the receivebeam at each second angle in the plurality of second angles, andcorresponding measured signal parameters at each second angle in theplurality of second angles.
 2. The method of claim 1, further comprisingrepeating the method of claim 1 for each of the at least one othernodes.
 3. The method of claim 1, wherein the plurality of first anglesand the plurality of second angles include angles in three-dimensionalspace.
 4. The method of claim 1, wherein the characteristics of thetransmit beam include beamforming parameters and transmit power.
 5. Themethod of claim 1, wherein the characteristics of the receive beaminclude beamforming parameters.
 6. The method of claim 1, wherein themeasured signal parameters include a power and multipath parameters ofthe signal received at the at least one other node.
 7. The method ofclaim 1, wherein the first node and the at least one other node are notline-of-sight.
 8. The method of claim 1, further comprising using the RFmap to compute two or more routes for communication between the firstnode and the at least one other node.
 9. The method of claim 8, whereinthe two or more routes include different beamforming parameters for thetransmit beams and the receive beams.
 10. The method of claim 1, whereinthe RF map includes one or more of a transmit beam angle, a receive beamangle, a transmit beam power, coding parameters, and modulationparameters.
 11. The method of claim 1, further comprising periodicallyupdating the RF map.
 12. The method of claim 11, where the RF map isupdated when traffic in the wireless network is below a predeterminedthreshold.
 13. The method of claim 1, wherein the first node includestwo or more antenna elements, and wherein the transmit beam is formedusing the two or more antenna elements.
 14. The method of claim 1,wherein the at least one other node includes two or more antennaelements, and wherein the receive beam is formed using the two or moreantenna elements.
 15. A system, comprising: a wireless network includinga plurality of nodes, wherein a respective node is configurable to: seta transmit beam to transmit a signal at a plurality of first angles; seta receive beam to scan a plurality of second angles and measure signalparameters of a received signal at each second angle in the plurality ofsecond angles; and a server configured to: for each node in the wirelessnetwork, configure a transmit beam of the node to transmit a signal atthe plurality of first angles; while the node is transmitting the signalat each first angle in the plurality of first angles, configure thereceive beam of at least one other node in the wireless network to scanthe plurality of second angles; and configure the at least one othernode to measure signal parameters of the signal at each second angle inthe plurality of second angles; and calculate an RF map based at leastin part on characteristics of the transmit beam at each first angle inthe plurality of first angles, corresponding characteristics of thereceive beam at each second angle in the plurality of second angles, andcorresponding measured signal parameters at each second angle in theplurality of second angles.
 16. The system of claim 15, wherein theplurality of first angles and the plurality of second angles includeangles in three-dimensional space.
 17. The system of claim 15, whereinthe characteristics of the transmit beam include beamforming parametersand transmit power.
 18. The system of claim 15, wherein thecharacteristics of the receive beam include beamforming parameters. 19.The system of claim 15, wherein the measured signal parameters include apower and multipath parameters of the signal received at the at leastone other nodes.
 20. The system of claim 15, wherein the node and the atleast one other node are not line-of-sight.
 21. The system of claim 15,wherein the server is configured to use the RF map to compute two ormore routes for communication between the node and the at least oneother node.
 22. The system of claim 21, wherein the two or more routesinclude different beamforming parameters for the transmit beams and thereceive beams.
 23. The system of claim 15, wherein the RF map includesone or more of a transmit beam angle, a receive beam angle, a transmitbeam power, coding parameters, and modulation parameters.
 24. The systemof claim 15, wherein the server is configured to periodically update theRF map.
 25. The system of claim 24, where the RF map is updated whentraffic in the wireless network is below a predetermined threshold. 26.The system of claim 15, wherein the node includes two or more antennaelements, and wherein the transmit beam is formed using the two or moreantenna elements.
 27. The system of claim 15, wherein the at least oneother node includes two or more antenna elements, and wherein thereceive beam is formed using the two or more antenna elements.